test(e2e): real-device Playwright messaging suite by thebentern · Pull Request #1121 · meshtastic/web

thebentern · 2026-06-15T23:37:32Z

What

A Playwright end-to-end suite that drives the real web app in Chromium
against real meshtasticd firmware over the HTTP phone API and verifies
text messaging in both directions across a real two-node mesh.

✓ connect over HTTPS → config handshake → messages view
✓ mesh → web   (peer node broadcasts; the browser renders it)
✓ web → mesh   (the browser sends; the peer node confirms receipt over the mesh)
- direct message  (skipped — PKI "Keys Mismatch", see Known limitations)

3 passed, 1 skipped (~26s)

How it works

Two meshtasticd sim nodes (Docker) mesh over the firmware's built-in UDP
multicast LAN transport (224.0.0.69:4403) — real firmware, real encryption,
distinct node numbers, no MQTT / no relay. The browser connects to Node A's
HTTPS phone API; an off-browser Python meshtastic peer drives/asserts
Node B over the TCP phone API. The same specs run against physical hardware
via E2E_DEVICE_MODE=hardware (the radio is the bridge).

   Playwright (headless Chromium)              Python peer (meshtastic lib)
   ── HTTPS phone API :9443 ──┐                ┌──── TCP phone API :4403 ────
                              ▼                ▼
                  ┌─────────────────┐  UDP multicast  ┌─────────────────┐
                  │  Node A (DUT)   │  224.0.0.69     │   Node B (peer) │
                  │ meshtasticd sim │◀─── mesh ──────▶│ meshtasticd sim │
                  └─────────────────┘                 └─────────────────┘

Builds on the firmware's own test foundation (the meshtasticd simulator and
the mcp-server/tests/mesh ReceiveCollector / wait_for patterns).

Run it

pnpm install
pnpm exec playwright install chromium
python -m venv e2e/peer/.venv && e2e/peer/.venv/bin/pip install -r e2e/peer/requirements.txt
pnpm test:e2e

CI: .github/workflows/e2e.yml (Linux, Docker sim). Full docs + env vars +
hardware mode: e2e/README.md.

🐞 Bugs surfaced by the suite

The suite immediately caught regressions from the #1050 SDK migration on main:

Fixed here — connect-on-save was broken. useConnections.connect()
resolved the connection id against a stale memoized connections closure, so
addConnectionAndConnect() reported unknown connection id and Save never
actually connected any HTTP/Serial/Bluetooth device. Now reads from the
live store (useDeviceStore.getState()). This affects real users, not just
the test.
Not fixed — ReferenceError: nodeDB is not defined in
apps/web/src/core/subscriptions.ts on every device-metrics telemetry packet
(the migration removed that store). Caught per-packet so messaging still
works, but it logs an error each time. The proper fix is a design call: route
device metrics into the SDK NodesClient vs. drop the handler.
Not fixed — direct messages are blocked by a PKI "Keys Mismatch": the
SDK's stored public key for the peer node does not match the key presented
during NodeInfo exchange, even with fresh sim nodes. The DM test is fixme
pending key-verification work.

Known limitations / notes

DMs are fixme (see Wrong Time-stamp in firmware 1.2.50 #3); broadcast already covers bidirectional messaging.
The SDK's sqlocal (SQLite/OPFS) store times out in headless Chromium and
falls back to in-memory; MessagesPage.waitReady() gates on the "Connected"
status so a send issued the instant the composer renders isn't silently
dropped.
Uses meshtastic/meshtasticd:daily-debian — the :latest tag (2.7.15)
predates the EnableUDP multicast feature, so sim nodes wouldn't mesh.

…tore addConnectionAndConnect() adds a connection and then connects to it in the same tick, but connect() resolved the id against the memoized `connections` closure, which is stale until the hook re-renders. The just-added id was therefore reported as an unknown connection id and Save silently never connected any HTTP/Serial/Bluetooth device. Read savedConnections from useDeviceStore.getState() so the lookup always sees the live store.

Drives the actual web app in Chromium against real meshtasticd firmware over the HTTP phone API and verifies text messaging in both directions across a two-node mesh. Nodes mesh over the firmware's built-in UDP multicast (224.0.0.69) with no MQTT/relay; distinct node numbers, real encryption. - Default backend: two Docker meshtasticd sim nodes (daily-debian). The same specs run against physical hardware via E2E_DEVICE_MODE=hardware. - An off-browser Python meshtastic peer (e2e/peer/peer.py) drives/asserts the non-browser node over the TCP phone API, mirroring firmware mcp-server tests. - Coverage: connect over HTTPS, mesh->web receive, web->mesh send. Direct messages are fixme'd (see below). CI workflow runs it on Linux. Bugs surfaced by the suite: - Fixed (prior commit): connect-on-save never connected (stale-closure id lookup in useConnections). - Not fixed: apps/web/src/core/subscriptions.ts throws 'ReferenceError: nodeDB is not defined' on every device-metrics telemetry packet (the #1050 migration removed that store); caught per-packet, so messaging still works. - Not fixed: direct messages are blocked by a PKI 'Keys Mismatch' (the SDK's stored peer public key != the key presented during NodeInfo exchange), seen even with fresh sim nodes.

vercel · 2026-06-15T23:37:38Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
web-test	Ready	Preview, Comment	Jun 16, 2026 1:01am

Copilot

Pull request overview

Adds a Playwright end-to-end suite that drives the real web app (Chromium) against real meshtasticd firmware nodes (Docker sim by default, hardware optional) to validate bidirectional text messaging over the HTTP(S) phone API + a two-node UDP-multicast mesh. Also includes a small but user-impacting fix to connection “connect-on-save” behavior in the web app.

Changes:

Introduces Playwright E2E infrastructure (config, scripts, CI workflow) and a documented real-device test harness.
Adds the E2E mesh topology (docker-compose + node configs), a Python TCP “peer” driver, and Playwright page objects + messaging specs.
Fixes useConnections.connect() to read the newly-added connection from the live store instead of a stale hook closure.

Reviewed changes

Copilot reviewed 20 out of 21 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`pnpm-lock.yaml`	Adds Playwright dependencies to the lockfile.
`playwright.config.ts`	New Playwright config wiring global setup/teardown, web server, and Chromium project settings.
`package.json`	Adds `test:e2e*` scripts and `@playwright/test` devDependency.
`e2e/tests/connect.spec.ts`	E2E “connect + reach messages view” smoke test.
`e2e/tests/messaging.broadcast.spec.ts`	E2E broadcast messaging tests (mesh→web and web→mesh).
`e2e/tests/messaging.direct.spec.ts`	Direct-message spec (currently `fixme` due to key mismatch behavior).
`e2e/README.md`	Full documentation for running the suite, env vars, and topology details.
`e2e/peer/requirements.txt`	Python peer dependency list (`meshtastic`).
`e2e/peer/peer.py`	Standalone Python peer for sending/receiving messages and reporting node number via TCP phone API.
`e2e/pages/ConnectionPage.ts`	Page object for driving the “Add Connection” dialog and connecting via HTTP(S).
`e2e/pages/MessagesPage.ts`	Page object for composer interactions and message assertions.
`e2e/fixtures/test.ts`	Shared Playwright fixtures for device info and page objects.
`e2e/fixtures/peer.ts`	TS wrapper around the Python peer process (send/recv/node-num).
`e2e/global-setup.ts`	Brings up mesh topology (docker mode) and waits for device + peer readiness.
`e2e/global-teardown.ts`	Tears down (or leaves running) the docker mesh after tests.
`e2e/device/docker-compose.yml`	Two-node `meshtasticd` sim topology for the suite.
`e2e/device/nodeA.yaml`	Node A config (HTTPS webserver + UDP multicast enabled).
`e2e/device/nodeB.yaml`	Node B config (UDP multicast enabled, no webserver).
`e2e/.gitignore`	Ignores Playwright artifacts and the Python venv/cache.
`apps/web/src/pages/Connections/useConnections.ts`	Fixes connect-on-save by reading the connection from the live store.
`.github/workflows/e2e.yml`	Adds CI workflow to run the E2E suite and upload the Playwright report.

Files not reviewed (1)

pnpm-lock.yaml: Generated file

+  const shouldDown = !!process.env.CI || process.env.E2E_DOCKER_DOWN === "1";
+  if (MODE !== "docker") return;
+  if (shouldDown) {
+    console.log("[e2e] tearing down meshtasticd mesh ...");
+    execSync(`docker compose -f ${COMPOSE_FILE} down -v`, { stdio: "inherit" });
+  } else {
+    console.log("[e2e] leaving meshtasticd mesh running (set E2E_DOCKER_DOWN=1 to stop).");
+  }


- waitForTcp(): destroy the probe socket on the error path so repeated connection failures don't accumulate sockets/FDs across the retry loop. - Don't remove the mesh containers in Playwright globalTeardown in CI — it raced the workflow's failure log capture. Teardown is now gated on E2E_DOCKER_DOWN only; CI dumps device logs on failure and tears the mesh down in a final always() workflow step.

apps/web/src/core/subscriptions.ts called nodeDB.addDeviceMetrics() on every device-metrics telemetry packet, but the #1050 migration removed that store — so it threw 'ReferenceError: nodeDB is not defined' on each telemetry packet (caught per-packet by the SDK's HandleFromRadio, so messaging still worked but the error spammed the console). Route device metrics into the SDK NodesClient via onTelemetryPacket instead — mirroring the existing position handler; the Node domain already carries a deviceMetrics field — and drop the dead app-side handler. Adds a NodesClient test covering the fold.

The direct-message fixme is a simulator limitation, not a web-app bug: the keyless meshtasticd sim nodes NAK a DM with NO_CHANNEL (routing error 6) — no Curve25519 keypair is provisioned/shared, and current firmware can't deliver a direct message without a per-node key / decryptable channel. The app surfaces this correctly (key-refresh dialog). Re-enable against hardware or once the sim provisions keys. Also: mark the nodeDB telemetry bug fixed and note the CI teardown change.

thebentern · 2026-06-16T00:13:10Z

Addressed the review feedback + the two bugs the suite surfaced:

Copilot review

waitForTcp() now destroy()s the probe socket on the error path (no FD accumulation across retries). 3ce46571
CI teardown no longer races the failure-log capture: Playwright teardown is gated on E2E_DOCKER_DOWN only, and the workflow dumps device logs on failure then tears the mesh down in a final always() step. 3ce46571

Bug 1 — ReferenceError: nodeDB is not defined (fixed) 3348db4d
subscriptions.ts called a node store the #1050 migration removed, throwing on every device-metrics telemetry packet. Routed device metrics into the SDK NodesClient via onTelemetryPacket (mirrors the position handler; Node.deviceMetrics already exists) and dropped the dead app-side handler. Added a NodesClient test; verified the error is gone end-to-end.

Bug 2 — direct messages (determined to be a simulator limitation, not a web bug) 3049235f
Root-caused: the DM send is NAK'd with NO_CHANNEL (routing error 6). The meshtasticd sim nodes never end up with a usable PKI keypair (no Curve25519 key provisioned/shared in NodeInfo), and current firmware can't deliver a DM without a per-node key / decryptable channel. The earlier "Keys Mismatch" was just the dialog's fixed copy. The app behaves correctly (raises the key-refresh dialog). The DM spec stays fixme with the accurate root cause; broadcast already covers bidirectional messaging. Re-enable against real hardware (nodes with PKI keys).

Suite still green: 3 passed, 1 skipped.

Followed up on the suggestion to provision keys in config.security: the keys ARE settable and persist (verified via admin), but on the native meshtasticd sim they don't sync to the node's owner / NodeInfo key — owner.public_key stays empty and the node keeps its MAC-derived num — so the two nodes never exchange keys. Combined with the firmware refusing non-PKI DMs ('Unknown public key for destination ... refusing to send legacy DM'), the DM is NAK'd with NO_CHANNEL. A firmware/sim limitation; DMs work on real hardware. Spec stays fixme.

Per the steer to research the firmware: PKI keygen is gated on a set LoRa region (NodeDB.cpp:3051) and the sim boots region-UNSET — setting lora.region via admin DOES make the nodes generate and exchange keys (verified both ways). But a PKI-encrypted DM still can't traverse the SimRadio: the PKC overhead exceeds its payload limit ('Payload size larger than compressed message allows! Send empty payload'), so the packet is truncated and the receiver NAKs NO_CHANNEL ('No suitable channel found for decoding, hash 0x0'). The firmware skips PKC under --sim (Router.cpp:730) for exactly this reason, but --sim also disables the config-file loading the web app needs, so they're mutually exclusive. DMs work on real hardware; spec stays fixme with this detail.

thebentern · 2026-06-16T01:01:02Z

Update on the DM fixme — dug into the firmware (thanks for the nudge). Two gates:

Region gate (fixable): PKI keygen is skipped while lora.region == UNSET (NodeDB.cpp:3051), and the sim boots region-UNSET. Setting lora.region via admin does make the nodes generate and exchange PKI keys — verified both nodes learn each other's public key.
SimRadio payload limit (the hard blocker): even with keys exchanged, a PKI DM can't traverse the SimRadio. The PKC overhead pushes the packet past the SimRadio payload limit (Payload size larger than compressed message allows! Send empty payload), so it's truncated and the receiver NAKs NO_CHANNEL (No suitable channel found for decoding, hash 0x0). The firmware deliberately skips PKC under --sim (Router.cpp:730) for exactly this reason — but --sim also disables config-file loading (Webserver/EnableUDP/MAC) the web app's HTTP API + UDP mesh need, so the two are mutually exclusive.

Net: DMs can't traverse the simulated radio in the config the web app requires; they work on real hardware (real LoRa carries PKC). DM spec stays fixme with this detail; broadcast still covers bidirectional messaging. Suite green (3 passed, 1 skipped).

danditomaso · 2026-06-17T12:16:46Z

@thebentern small nitpick, can details about this get added to the readme.md in the apps/web folder? I think it's important usrre if this repo understand how this testing works

thebentern added 2 commits June 15, 2026 18:36

Copilot AI review requested due to automatic review settings June 15, 2026 23:37

Copilot started reviewing on behalf of thebentern June 15, 2026 23:38 View session

vercel Bot deployed to Preview – web-test June 15, 2026 23:38 View deployment

Copilot AI reviewed Jun 15, 2026

View reviewed changes

thebentern added 3 commits June 15, 2026 19:12

vercel Bot deployed to Preview – web-test June 16, 2026 00:13 View deployment

vercel Bot deployed to Preview – web-test June 16, 2026 00:30 View deployment

vercel Bot deployed to Preview – web-test June 16, 2026 01:01 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(e2e): real-device Playwright messaging suite#1121

test(e2e): real-device Playwright messaging suite#1121
thebentern wants to merge 7 commits into
mainfrom
e2e-messaging

thebentern commented Jun 15, 2026

Uh oh!

vercel Bot commented Jun 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

thebentern commented Jun 16, 2026

Uh oh!

thebentern commented Jun 16, 2026

Uh oh!

danditomaso commented Jun 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

thebentern commented Jun 15, 2026

What

How it works

Run it

🐞 Bugs surfaced by the suite

Known limitations / notes

Uh oh!

vercel Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

thebentern commented Jun 16, 2026

Uh oh!

thebentern commented Jun 16, 2026

Uh oh!

danditomaso commented Jun 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vercel Bot commented Jun 15, 2026 •

edited

Loading